Lip-Reading Driven Deep Learning Approach for Speech Enhancement

This paper proposes a novel lip-reading driven deep learning framework for speech enhancement. The approach leverages the complementary strengths of both and analytical acoustic modeling (filtering-based approach) as compared to benchmark approaches that rely only on learning. proposed audio-visual (AV) enhancement operates at two levels. In first level, based regression model is employed. second approximated clean-audio features are exploited, using an enhanced, visually-derived Wiener filter (EVWF), estimating clean audio power spectrum. Specifically, stacked long-short-term memory (LSTM) designed temporal visual (i.e., lip reading), by considering range prior frames. For spectrum estimation, new filterbank-domain EVWF formulated, which exploits estimated features. with conventional spectral subtraction log-minimum mean-square error methods ideal AV mapping LSTM approaches. potential evaluated under four different dynamic real-world scenarios [cafe, street junction, public transport, pedestrian area] SNR levels (ranging from low high SNRs) grid ChiME3 corpora. objective testing, perceptual evaluation quality used evaluate restored speech. subjective standard mean-opinion-score method inferential statistics. Comparative simulation results demonstrate significant improvements in terms intelligibility. Ongoing work aimed enhancing accuracy generalization capability model, contextual integration cues, leading context-aware, autonomous